Using Pupil Size as a Measure of Cognitive Workload in Video-Based Eye-Tracking Studies

175 words Text (excluding references and figure captions): 5526 References: 19 entries Figures: 9 Tables: 0 Abstract Pupil size is known to quickly adapt to changes in the luminance within the visual field and the observer’s cognitive workload while performing a visual task. Pupil size is rarely analyzed in eye-movement studies although it is measured by most video-based eyetracking systems. This is mainly due to two problems: First, the dependence of pupil size on luminance across the display and second, the distortion of pupil-size data by eye movements: The size of the pupil as measured by the eye-tracker camera depends on the subject's gaze angle. In the present study, we introduce and develop measures and heuristics to estimate luminance-based changes in pupil size. Moreover, we present a neural-network based pupil calibration interface for eye-tracking systems, which is capable of almost completely eliminating the geometry-based distortion of pupil-size data. Finally, we compare the effects of cognitive workload and display luminance on pupil dilation and investigate the interaction of these two factors. The results of our study facilitate the use of pupil dilation as a reliable and inexpensive indicator of a subject's cognitive workload.Pupil size is known to quickly adapt to changes in the luminance within the visual field and the observer’s cognitive workload while performing a visual task. Pupil size is rarely analyzed in eye-movement studies although it is measured by most video-based eyetracking systems. This is mainly due to two problems: First, the dependence of pupil size on luminance across the display and second, the distortion of pupil-size data by eye movements: The size of the pupil as measured by the eye-tracker camera depends on the subject's gaze angle. In the present study, we introduce and develop measures and heuristics to estimate luminance-based changes in pupil size. Moreover, we present a neural-network based pupil calibration interface for eye-tracking systems, which is capable of almost completely eliminating the geometry-based distortion of pupil-size data. Finally, we compare the effects of cognitive workload and display luminance on pupil dilation and investigate the interaction of these two factors. The results of our study facilitate the use of pupil dilation as a reliable and inexpensive indicator of a subject's cognitive workload. Using Pupil Size as a Measure of Cognitive Workload in Video-Based Eye-Tracking Studies Cognitive workload is an important concept for both the study of human cognition and the evaluation of human-machine interfaces such as head-up displays in cars or air-traffic controllers’ workstations. There are several common methods for measuring cognitive workload: galvanic skin response, heart rate, and electroencephalography (e.g., Kramer, 1991; O’Donnell & Eggemeier, 1986; Wilson, 2001). Rather than taking cognitive workload measurements, many researchers evaluate interfaces by analyzing users’ eye movements during task completion (e.g., Goldberg & Kotval, 1999). Gaze trajectories can indicate difficulties that users encounter with certain parts of the interface and point out inappropriate spatial arrangement of interface components. Interestingly, almost all of these studies use video-based eye trackers, which means that they routinely disregard an indicator of cognitive workload that they receive as a “byproduct”, namely the size of the user’s pupil. It is well known from a variety of studies that an observer’s pupils dilate with increasing cognitive workload being imposed (see Kahneman, 1973). This effect has been demonstrated for tasks such as mental arithmetic (Hess, 1965), sentence comprehension (Just & Carpenter, 1993), letter matching (Beatty & Wagoner, 1978), and visual search (Porter, Troscianko & Gilchrist, 2007). Besides cognitive workload, also emotional factors such as the emotional content of written words influence pupil size (Võ et al., 2008). However, in typical laboratory tasks, emotional influence can easily be reduced so that it does not significantly interfere with cognitive workload measurement. Unfortunately, such control is more difficult to achieve for the third factor determining pupil size the illumination of the observer’s visual field (Reeves, 1920). If changes in illumination need to occur during experimental sessions, we can expect substantial interference with the use of pupil size as a measure of cognitive workload. To reliably measure workload, we have to account for such changes in illumination (Nakayama, Yasuike & Shimizu, 1990). Unfortunately, the only current approach to this problem is to control the illumination of the display (e.g., Porter, Troscianko & Gilchrist 2002; 2007), which is not always possible when evaluating interfaces. Furthermore, scientists face a technical problem: Since participants usually move their eyes during experiments, their pupils assume different angles and distances toward the monitoring camera of the eye tracker. This, in turn, means that the size of the pupil as measured by the system the number of pixels in the camera image covered by the pupil or an ellipse fitted to it – varies with the participant’s gaze angle. This effect is especially strong if the camera is located below the eye, which is the case for most remote eye trackers (using desktop cameras) and head-mounted systems. Consequently, these systems report considerably larger average pupil size while subjects fixate targets at the bottom of the screen than when their gaze is near the top of the screen. This effect makes it impossible to measure pupil size consistently in tasks involving eye movements. Even when only the average or maximum pupil size during a trial is of interest, any systematic difference in the distribution of fixation positions across experimental conditions would invalidate the pupil size measurements. To tackle these problems, the present study provides methods for both dissociating the pupil’s responses to light versus cognitive workload, as well as for deriving a gaze-position invariant measure of pupil size. In Experiment 1, the basic pupillary response to different levels of luminance on a standard CRT screen was obtained. Based on the results, we propose the pupil constriction index as a suitable measure for the pupil’s response to screen luminance. Experiment 2 examined the effect of stimulus color – its red, green, and blue components displayed by the monitor – and eccentricity on pupil size. The results are used to develop heuristics for estimating the pupil’s light response to any given display. Moreover, we introduced a neural-network based pupil calibration interface and evaluated it empirically in Experiment 3. It is shown that this technique greatly reduces the distortion of pupil size measurement by gaze shifts and thereby provides a valid pupil size measure for tasks involving eye movements. In the final Experiment 4, the neural-network interface is used to analyze cognitive workload in a monitoring task. The findings suggest that illumination and cognitive workload control pupil size in distinct ways, which needs to be considered when dissociating these two factors. Experiment 1: Pupillary Response to Changes in Luminance on a Computer Monitor While the response of the pupil to changes in illumination has already been measured in previous studies (e.g., Reeves, 1920), the main purpose of Experiment 1 was to derive useful measures for estimating the pupillary response to changes in luminance on a standard CRT computer monitor. Another aim of this experiment was to determine the time course of this response. Method Participants. Ten students from the University of Massachusetts at Boston were tested individually. All participants had normal or corrected-to-normal vision. They were naïve with respect to the purpose of the study and were paid for their participation. Apparatus. Eye movements were recorded with the SR Research Ltd. EyeLink-II system, which operates at a sampling rate of 500 Hz and measures a participant’s gaze position with an average error of less than 0.5 degrees of visual angle. Stimuli were presented on a calibrated 19-inch Dell Trinitron CRT monitor with a refresh rate of 85 Hz and a screen resolution of 1024 by 768 pixels (CIE chromaticity values: red: x = 0.625, y = 0.340; green: x = 0.275, y = 0.605; blue: x = 0.150, y = 0.065; color temperature: 9300 K). The subjects were seated at an eye-screen distance of 50 cm. Materials. The stimulus displays showed a plus sign (approximately 1° in diameter) centered on a gray disc (28° in diameter) on a black background (0.2 cd/m). The disc assumed one of 15 different luminance levels (0.2, 5.2, 10.2, ..., 70.2 cd/m). The plus sign was white for disc luminance below 35 cd/m and was black otherwise. Procedure. Prior to the experiment, subjects performed a 9-point calibration procedure. The subsequent experiment consisted of 150 trials, which presented each disc luminance level ten times. These trials were presented in randomized order. The subjects started each trial by looking at a central drift correction marker – a white marker on a black background (0.2 cd/m) and pressing a button on a game pad. Every trial lasted for 12 seconds or until the subjects shifted their gaze away from the plus marker by more than 1.5° of visual angle (gaze error). Trials with gaze error were presented again later in the experiment, and the data recorded during their first presentation were excluded from analysis.